MULTIMEDIA ELEMENTS AND EMOTIONAL PROCESSES Multimedia , Stories , and Emotion -

There are theoretical models concerning Multimedia and cognitive and motivational processes. However, such models are missing for emotional processes. In order to develop such a model, studies from literary text research were analyzed. Literary text research was chosen as a starting point because Multimedia environments often contain texts (or stories) with an emotional quality. Analyzing and optimizing such texts or stories have a long tradition within this kind of research. The resulting model relates Multimedia elements, story elements (e.g., foregrounding), perceptualcognitive-emotional processes (e.g., presence), and emotional outcomes (e.g., pleasantness) to each other. Theoretical implications concern the expansion of the model in respect to other Multimedia elements and to the integration with existing cognitive and motivational approaches. Methodological implications focus on suggestions for measurement development. Practical implications deal with Multimedia design and dramaturgical e-learning strategies. Multimedia, Stories, and Emotion An Integrated Model for Research and Design Multimedia elements are fascinating people in computer games or web environments, not only because of their power to influence cognitive and motivational processes, but especially because of their capability to stimulate emotions, like fear or pleasure. How to arouse emotions with Multimedia represents an issue in research which is usually related to computer games. There is tremendous research on how computer games can produce emotions like, for example, aggression (e.g., Singer & Singer, 2002). However, this kind of research focuses mainly on negative emotions (e.g., anger) and on general long-term effects of different games or media settings (including TV). It shows little about which specific Multimedia elements produce positive influences on short-term emotional processes (e.g., Bryant & Zillmann, 2002). To identify short-term effects on emotional processes is important because it can be assumed that long-term effects are built when continuously short-term effects are cumulated. In addition, it can be assumed that short-term effects can more easily be used for educational purposes (e.g., for establishing emotionally sound learning environments for children). Identifying short-term effects of Multimedia elements was done in the field of cognitive and motivational processes based on sophisticated theory building and related empirical research (e.g., Astleitner & Wiesner, 2004; Keller, 1999; Malone & Lepper, 1987; Mayer, 2001). However, such activities are lacking in respect to emotional processes. For example, recent comprehensive reviews of Multimedia effects neglected entirely the emotional perspective (e.g., Hede, 2002). Indeed Astleitner and Leutner (2000) presented a research-based approach dealing with the question of how computer-based environments can be made more emotionally sound. But, this approach is focusing on instructional elements on a macro-level (referring to emotional outcomes), and not dealing with Multimedia elements on a micro-level (including emotional processes). Also, the "emotioneering"-techniques from Freeman (2003) do not improve the situation, because they are not related to research, but to the practice of game design. Such techniques induce, for example, a computer player to identify with the role he or she is playing. The over 300 distinct techniques include, for example, ways to give emotional depth to a non-player character, even if the player has just one line of dialogue. Finding suitable basic research dealing with Multimedia and emotion can be based on a definition of Multimedia. According to Jonassen (2000, p. 207), the commonality among these definitions "involves the integration of more than one medium into some form of communication" and "refers to the integration of media such as text, sound, graphics, animation, video, imaging, and spatial modeling into a computer system". Referring to this definition, a first step of research can be to relate each element of Multimedia to emotional processes, and a second step, to combine some elements and analyze their joint effects on emotional processes achieving finally a comprehensive model of different Multimedia effects on emotion. There is, for example, research on sound (e.g., Bradley & Lang, 2000), on music (e.g., Juslin & Sloboda, 2001), and on video (e.g., Cennamo, 1993) and emotional processes. However, the way of establishing a first research-based link from Multimedia elements to emotional processes in this paper is to focus on text and especially on literary text research. There are several reasons for this decision: a) one of the most frequently stated reasons for reading literary text is to experience emotions or emotional processes, so there is a strong relationship between literary text and emotion (see, for example, fiction with high emotional appeal from Gass, 1999 or Obermayr, 1998); b) the main elements in many Multimedia environments (e.g., learning environments on CD-ROMS or on the Internet) are texts, because texts represent the major source of information and handling of texts do not require sophisticated technical equipment (Jonassen, 1999); c) there is a long tradition in research on literary text and emotion what increases the chance to identify a highly elaborated network of variables which can build a basis for Multimedia research and design (e.g., Oxenhandler, 1988); and d) there are reviews concerning media in general and emotions, however, they do not focus on literary text and/or Multimedia (e.g., Schwab, 2001). Based on literary text research, the following research questions should be answered within this study: a) Which elements of texts do trigger emotional processes? For analyzing literary texts, the following aspects are usually considered: theme, character, plot, structure, setting, point of view, language and style, etc. These aspects must be evaluated in their capacity to produce emotions in Multimedia based on empirical literary text research. b) Which emotional processes are triggered? According to Kleinginna and Kleinginna (1981, p. 355) emotions are "a complex set of interactions among subjective and objective factors, mediated by neural/hormonal systems, which can: (a) give rise to affective experiences such as feelings of arousal, pleasure/displeasure; (b) generate cognitive processes such as emotionally relevant perceptual effects, appraisals, labeling processes; (c) activate widespread physiological adjustments to the arousing conditions; and (d) lead to behavior that is often, but not always, expressive, goal directed, and adaptive". Emotional processes are based on the acceptance or rejection of objects and facts and have an evaluational relation to the world (Kuhl, 1986). Emotions, although they interact with cognitive and motivational processes, are considered in research as a unique component of human mental states, experiences, and behavioral expressions (e.g., Frijda, 1986). Emotions may initiate, terminate, or disrupt information processing. They may result in selective information processing or they may organize recall. "Feelings" are the "conscious recognition of emotions" (Greenberg & Snell, 1997, p. 96). Emotional processes are interactive and numerous. It is the task of this paper to select these processes which turned out to be important in research and which are relevant for designing Multimedia environments. c) How can emotionally relevant elements of texts be used and implemented within Multimedia environments in order to improve their capacity to affect emotions? Such Multimedia environments allow for the electronically integrated display and user control of a variety of media formats and information types, including motion video and film, still photographs, text, graphics, animation, sound, numbers, and data. The resulting experience for the user is a multidimensional, multi-sensory interweave of self-directed reading, viewing, listening, and interacting, through activities such as exploring, searching, manipulating, writing, linking, creating, juxtaposing, or editing (e.g., Wilson 1992). Within this paper, however, only texts and their emotional capacities are analyzed. For example, from text comprehension research it is well known that learning from text and learning from, for example, pictures are qualitatively different ways of constructing mental models. While text comprehension is an indirect transformation between a text as a symbolic representation and the mental model as an analog representation, picture comprehension is a direct mapping of the picture as an analog representation onto the mental model as an analog representation by establishing an analogy between the visual information and the corresponding mental model (e.g., Chun & Plass, 1997). There exists no such comparable research for emotional processes. That is the reason why exclusively texts are analyzed in a first step of research. All the questions should be answered on the basis of a narrative literature review concerning results of empirical studies. This literature review is based on some general assumptions about emotions and media (e.g., Mangold, Unz, & WinterhoffSpurk, 2001). These assumptions are: Confrontations with media contents produce different intensities of emotional experiences. These emotions refer to situations situated within the media contents but also to real life situations which are, to some extent, similar to the media contents. Emotional experiences can be related to one, but also to different multiple emotions. Emotions can change rapidly in time. Finally, different people experience different emotions if confronted with the same media contents. According to a taxonomy of literature reviews from Cooper and Hedges (1994), the focus of the review concerning literary text research and emotion was on practices or applications in the field of reading, on the identification and integration of central issues, on a representative (but not exhaustive) coverage of literature, on a conceptual organization of research results, and on Multimedia researchers in the field of education and psychology as target group. Despite this anchoring in existing empirical research, it has to be stressed that the resulting review do not represent definite generalizations of empirically well-founded research results for Multimedia designers, but nothing more than hypothetical prescriptions which should guide researchers and practitioners in their daily work concerning research on and design of emotionally sound Multimedia. Although many of these hypothetical prescriptions refer to results found in empirical research, they also have a speculative element delivering an inspiring force for future Multimedia research and design. For that purpose, it is not useful to distinguish between empirically well-founded and not well-founded results. All results, even the ones related with some empirical evidence, should be seen as hypotheses, because they are part of a systematic and mingled framework which has not been tested yet in the field of Multimedia. A Model of Multimedia, Story Elements, and Emotions Figure 1 shows the variables which relates Multimedia elements to emotional output variables based on literary text research results. The identification of these variables was based on research from Alfes (1995), Appel, Koch, Schreier, and Groeben (2002), De Vega, Leon, and Diaz (1996), Lombard and Ditton (1997), Oatley (1994), and Miall and Kuiken (1994, 2002). Relations between these variables are hypothesized related to research from Dijkstra, Zwaan, Graesser, and Magliano (1994), Gygax, Oakhill, and Garnham (2003), Kneepkens and Zwaan (1994), Levorato (2003), Sadoski, Goetz, and Rodriguez (2000), and Van Reekum (2000). These research results were used to build a model of Multimedia, story elements, and emotions. Within a first step, important components from research were selected. Within a next step, their position within the relationship between Multimedia elements and emotional outcomes were fixed. Then, research results were used to connect components as hypotheses. Finally, the model was evaluated (e.g., by proofing for inconsistencies) and calibrated (e.g., by improving definitions of